Summary

• Fix StratifiedKFold shuffle to ensure independent per-class shuffling when shuffle=True.
• Use check_random_state(self.random_state) to derive per-class RNG seeds so equal-sized classes do not receive coordinated permutations.
• Update docstring for StratifiedKFold to clarify shuffle behavior; add regression tests.
• Link to Issue: StratifiedKFold shuffle shuffles only batch order; within-stratum permutations are coordinated across classes (bug) #21.

Reproduction Steps

import numpy as np
from sklearn.model_selection import StratifiedKFold

y = np.array([0]*20 + [1]*20)
X = np.zeros_like(y)

skf0 = StratifiedKFold(n_splits=5, shuffle=True, random_state=0)
skf1 = StratifiedKFold(n_splits=5, shuffle=True, random_state=1)

idx0 = np.where(y == 0)[0]
idx1 = np.where(y == 1)[0]

perm0_c0, perm0_c1 = [], []
for _, test_idx in skf0.split(X, y):
    perm0_c0.extend([np.where(idx0 == i)[0][0] for i in test_idx if y[i] == 0])
    perm0_c1.extend([np.where(idx1 == i)[0][0] for i in test_idx if y[i] == 1])

perm1_c0, perm1_c1 = [], []
for _, test_idx in skf1.split(X, y):
    perm1_c0.extend([np.where(idx0 == i)[0][0] for i in test_idx if y[i] == 0])
    perm1_c1.extend([np.where(idx1 == i)[0][0] for i in test_idx if y[i] == 1])

print('seed=0 class0 permutation:', perm0_c0)
print('seed=0 class1 permutation:', perm0_c1)
print('seed=1 class0 permutation:', perm1_c0)
print('seed=1 class1 permutation:', perm1_c1)
print('perm0_c0 == perm0_c1?', perm0_c0 == perm0_c1)
print('perm1_c0 == perm1_c1?', perm1_c0 == perm1_c1)

Observed Failure (pre-fix)

• Prior to this change, on the target branch, the above script printed True for both equality checks, indicating coordinated permutations across classes (identical within-stratum shuffles) even with different seeds. There is no exception; the failure is behavioral and reproducibility-related.
• Example (abridged) output before fix:

seed=0 class0 permutation: [7, 18, 5, 10, 0, ...]
seed=0 class1 permutation: [7, 18, 5, 10, 0, ...]
seed=1 class0 permutation: [13, 2, 16, 6, 9, ...]
seed=1 class1 permutation: [13, 2, 16, 6, 9, ...]
perm0_c0 == perm0_c1? True
perm1_c0 == perm1_c1? True

Fix and Post-Fix Behavior

• After this fix, per-class permutations are independent:

perm0_c0 == perm0_c1? False
perm1_c0 == perm1_c1? False

• New tests added:
  • test_stratifiedkfold_shuffle_independent_per_class_permutation
  • test_stratifiedkfold_shuffle_different_seeds_change_permutations
  • test_stratifiedkfold_no_shuffle_unchanged

Local Test Logs

• Environment: conda-forge micromamba (AArch64), Python 3.8, NumPy 1.21.6, SciPy 1.7.3, Cython 0.29.36.
• Commands executed (no CI):

python setup.py build_ext -i
pip install -e . --no-build-isolation
pytest -q sklearn/model_selection/tests/test_split.py::test_shuffle_stratifiedkfold_reproducibility
pytest -q sklearn/model_selection/tests/test_split.py::test_stratifiedkfold_shuffle_independent_per_class_permutation
pytest -q sklearn/model_selection/tests/test_split.py::test_stratifiedkfold_shuffle_different_seeds_change_permutations
pytest -q sklearn/model_selection/tests/test_split.py::test_stratifiedkfold_no_shuffle_unchanged

• Results: all targeted tests passed locally. No CI run was performed.

Notes

• This change intentionally alters split assignments for equal-sized classes when shuffle=True and random_state is an integer, to align with documentation and expected semantics (shuffle within each stratification independently).